{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SEC Form 4 Filings: Who is Buying and Who is Selling?\n",
    "\n",
    "Insider transaction reporting via SEC Form 4 plays an important role in market transparency by forcing corporate officers, directors, and large shareholders to disclose their stock trades within two business days of execution. Because Form 4s contain rich, structured information—transaction dates, security details, number of shares, prices, and the insider’s relationship to the issuer—systematically extracting and aggregating these filings enables analysts and investors to detect patterns such as confidence buys, opportunistic sales, or hedging activity long before quarterly disclosures arrive. \n",
    "\n",
    "Automating Form 4 extraction addresses challenges of inconsistent XML/TXT layouts, varying footnote structures, and high filing volumes, ensuring that trading signals derived from insider behavior are captured accurately, updated in real time, and integrated seamlessly into quantitative models and trading dashboards. While it is possible to manually extract from the XML data, this can be a time consuming and error prone process. We will show how we can use LlamaExtract to do this automatically without having to parse the XML data.\n",
    "\n",
    "## Dow Jones Industrial Average Companies\n",
    "\n",
    "Dow Jones Industrial Average (DJIA) is a stock market index that consists of 30 large companies listed on the New York Stock Exchange and the NASDAQ and is considered a good proxy for the overall US stock market. For this exercise, we will extract the insider transactions for all the companies in the DJIA. Let's first get the list of tickers in the Dow Jones Industrial Average using Wikipedia."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pandas in /Users/neerajprad/workspace/llama_cloud_services/.venv/lib/python3.11/site-packages (2.2.3)\n",
      "Requirement already satisfied: numpy>=1.23.2 in /Users/neerajprad/workspace/llama_cloud_services/.venv/lib/python3.11/site-packages (from pandas) (2.0.2)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /Users/neerajprad/workspace/llama_cloud_services/.venv/lib/python3.11/site-packages (from pandas) (2.9.0.post0)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/neerajprad/workspace/llama_cloud_services/.venv/lib/python3.11/site-packages (from pandas) (2025.2)\n",
      "Requirement already satisfied: tzdata>=2022.7 in /Users/neerajprad/workspace/llama_cloud_services/.venv/lib/python3.11/site-packages (from pandas) (2025.2)\n",
      "Requirement already satisfied: six>=1.5 in /Users/neerajprad/workspace/llama_cloud_services/.venv/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas) (1.17.0)\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.1.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Requirement already satisfied: lxml in /Users/neerajprad/workspace/llama_cloud_services/.venv/lib/python3.11/site-packages (5.4.0)\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.1.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install pandas\n",
    "%pip install lxml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['MMM',\n",
       " 'AXP',\n",
       " 'AMGN',\n",
       " 'AMZN',\n",
       " 'AAPL',\n",
       " 'BA',\n",
       " 'CAT',\n",
       " 'CVX',\n",
       " 'CSCO',\n",
       " 'KO',\n",
       " 'DIS',\n",
       " 'GS',\n",
       " 'HD',\n",
       " 'HON',\n",
       " 'IBM',\n",
       " 'JNJ',\n",
       " 'JPM',\n",
       " 'MCD',\n",
       " 'MRK',\n",
       " 'MSFT',\n",
       " 'NKE',\n",
       " 'NVDA',\n",
       " 'PG',\n",
       " 'CRM',\n",
       " 'SHW',\n",
       " 'TRV',\n",
       " 'UNH',\n",
       " 'VZ',\n",
       " 'V',\n",
       " 'WMT']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "url = \"https://en.wikipedia.org/wiki/Dow_Jones_Industrial_Average\"\n",
    "parsed = pd.read_html(url, header=0)\n",
    "dow = next(t for t in parsed if \"Symbol\" in t.columns)\n",
    "DJIA_TICKERS = dow[\"Symbol\"].tolist()\n",
    "DJIA_TICKERS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see an example of a Form 4 filing for MMM. Disregard how we got the URL below, we will show how to get the URL for all the Form 4 filings in the utility function below.\n",
    "\n",
    "🖋: We are using `hishel` to cache the responses from the SEC website to avoid being rate limited or blocked."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: hishel in /Users/neerajprad/workspace/llama_cloud_services/.venv/lib/python3.11/site-packages (0.1.2)\n",
      "Requirement already satisfied: httpx>=0.28.0 in /Users/neerajprad/workspace/llama_cloud_services/.venv/lib/python3.11/site-packages (from hishel) (0.28.1)\n",
      "Requirement already satisfied: anyio in /Users/neerajprad/workspace/llama_cloud_services/.venv/lib/python3.11/site-packages (from httpx>=0.28.0->hishel) (3.7.1)\n",
      "Requirement already satisfied: certifi in /Users/neerajprad/workspace/llama_cloud_services/.venv/lib/python3.11/site-packages (from httpx>=0.28.0->hishel) (2025.4.26)\n",
      "Requirement already satisfied: httpcore==1.* in /Users/neerajprad/workspace/llama_cloud_services/.venv/lib/python3.11/site-packages (from httpx>=0.28.0->hishel) (1.0.9)\n",
      "Requirement already satisfied: idna in /Users/neerajprad/workspace/llama_cloud_services/.venv/lib/python3.11/site-packages (from httpx>=0.28.0->hishel) (3.10)\n",
      "Requirement already satisfied: h11>=0.16 in /Users/neerajprad/workspace/llama_cloud_services/.venv/lib/python3.11/site-packages (from httpcore==1.*->httpx>=0.28.0->hishel) (0.16.0)\n",
      "Requirement already satisfied: sniffio>=1.1 in /Users/neerajprad/workspace/llama_cloud_services/.venv/lib/python3.11/site-packages (from anyio->httpx>=0.28.0->hishel) (1.3.1)\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.1.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install hishel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<SEC-DOCUMENT>0001127602-25-005147.txt : 20250218\n",
      "<SEC-HEADER>0001127602-25-005147.hdr.sgml : 20250218\n",
      "<ACCEPTANCE-DATETIME>20250218111341\n",
      "ACCESSION NUMBER:\t\t0001127602-25-005147\n",
      "CONFORMED SUBMISSION TYPE:\t4\n",
      "PUBLIC DOCUMENT COUNT:\t\t1\n",
      "CONFORMED PERIOD OF REPORT:\t20250207\n",
      "FILED AS OF DATE:\t\t20250218\n",
      "DATE AS OF CHANGE:\t\t20250218\n",
      "\n",
      "REPORTING-OWNER:\t\n",
      "\n",
      "\tOWNER DATA:\t\n",
      "\t\tCOMPANY CONFORMED NAME:\t\t\tChavez Rodriguez Beatriz Karina\n",
      "\t\tCENTRAL INDEX KEY:\t\t\t0001890011\n",
      "\t\tORGANIZATION NAME:           \t\n",
      "\n",
      "\tFILING VALUES:\n",
      "\t\tFORM TYPE:\t\t4\n",
      "\t\tSEC ACT:\t\t1934 Act\n",
      "\t\tSEC FILE NUMBER:\t001-03285\n",
      "\t\tFILM NUMBER:\t\t25633441\n",
      "\n",
      "\tMAIL ADDRESS:\t\n",
      "\t\tSTREET 1:\t\t3M COMPANY\n",
      "\t\tSTREET 2:\t\t3M CENTER BLDG 220-09-E-02\n",
      "\t\tCITY:\t\t\tST. PAUL\n",
      "\t\tSTATE:\t\t\tMN\n",
      "\t\tZIP:\t\t\t55144-1000\n",
      "\n",
      "ISSUER:\t\t\n",
      "\n",
      "\tCOMPANY DATA:\t\n",
      "\t\tCOMPANY CONFORMED NAME:\t\t\t3M CO\n",
      "\t\tCENTRAL INDEX KEY:\t\t\t0000066740\n",
      "\t\tSTANDARD INDUSTRIAL CLASSIFICATION:\tSURGICAL & MEDICAL INSTRUMENTS & APPARATUS [3841]\n",
      "\t\tORGANIZATION NAME:           \t08 Industrial Applications and Services\n",
      "\t\tIRS NUMBER:\t\t\t\t410417775\n",
      "\t\tSTATE OF INCORPORATION:\t\t\tDE\n",
      "\t\tFISCAL YEAR END:\t\t\t1231\n",
      "\n",
      "\tBUSINESS ADDRESS:\t\n",
      "\t\tSTREET 1:\t\t3M CENTER\n",
      "\t\tSTREET 2:\t\tBLDG. 220-13E-26A\n",
      "\t\tCITY:\t\t\tST PAUL\n",
      "\t\tSTATE:\t\t\tMN\n",
      "\t\tZIP:\t\t\t55144-1000\n",
      "\t\tBUSINESS PHONE:\t\t6517331474\n",
      "\n",
      "\tMAIL ADDRESS:\t\n",
      "\t\tSTREET 1:\t\t3M CENTER\n",
      "\t\tSTREET 2:\t\tBLDG. 220-13E-26A\n",
      "\t\tCITY:\t\t\tST. PAUL\n",
      "\t\tSTATE:\t\t\tMN\n",
      "\t\tZIP:\t\t\t55144-1000\n",
      "\n",
      "\tFORMER COMPANY:\t\n",
      "\t\tFORMER CONFORMED NAME:\tMINNESOTA MINING & MANUFACTURING CO\n",
      "\t\tDATE OF NAME CHANGE:\t19920703\n",
      "</SEC-HEADER>\n",
      "<DOCUMENT>\n",
      "<TYPE>4\n",
      "<SEQUENCE>1\n",
      "<FILENAME>form4.xml\n",
      "<DESCRIPTION>PRIMARY DOCUMENT\n",
      "<TEXT>\n",
      "<XML>\n",
      "<?xml version=\"1.0\"?>\n",
      "<ownershipDocument>\n",
      "\n",
      "    <schemaVersion>X0508</schemaVersion>\n",
      "\n",
      "    <documentType>4</documentType>\n",
      "\n",
      "    <periodOfReport>2025-02-07</periodOfReport>\n",
      "\n",
      "    <issuer>\n",
      "        <issuerCik>0000066740</issuerCik>\n",
      "        <issuerName>3M CO</issuerName>\n",
      "        <issuerTradingSymbol>MMM</issuerTradingSymbol>\n",
      "    </issuer>\n",
      "\n",
      "    <reportingOwner>\n",
      "        <reportingOwnerId>\n",
      "            <rptOwnerCik>0001890011</rptOwnerCik>\n",
      "            <rptOwnerName>Chavez Rodriguez Beatriz Karina</rptOwnerName>\n",
      "        </reportingOwnerId>\n",
      "        <reportingOwnerAddress>\n",
      "            <rptOwnerStreet1>3M CENTER</rptOwnerStreet1>\n",
      "            <rptOwnerStreet2></rptOwnerStreet2>\n",
      "            <rptOwnerCity>ST. PAUL</rptOwnerCity>\n",
      "            <rptOwnerState>MN</rptOwnerState>\n",
      "            <rptOwnerZipCode>55144</rptOwnerZipCode>\n",
      "            <rptOwnerStateDescription></rptOwnerStateDescription>\n",
      "        </reportingOwnerAddress>\n",
      "        <reportingOwnerRelationship>\n",
      "            <isOfficer>1</isOfficer>\n",
      "            <officerTitle>Group President</officerTitle>\n",
      "        </reportingOwnerRelationship>\n",
      "    </reportingOwner>\n",
      "\n",
      "    <aff10b5One>0</aff10b5One>\n",
      "\n",
      "    <nonDerivativeTable>\n",
      "        <nonDerivativeTransaction>\n",
      "            <securityTitle>\n",
      "                <value>Common Stock</value>\n",
      "            </securityTitle>\n",
      "            <transactionDate>\n",
      "                <value>2025-02-07</value>\n",
      "            </transactionDate>\n",
      "            <transactionCoding>\n",
      "                <transactionFormType>4</transactionFormType>\n",
      "                <transactionCode>M</transactionCode>\n",
      "                <equitySwapInvolved>0</equitySwapInvolved>\n",
      "            </transactionCoding>\n",
      "            <transactionTimeliness></transactionTimeliness>\n",
      "            <transactionAmounts>\n",
      "                <transactionShares>\n",
      "                    <value>3569</value>\n",
      "                </transactionShares>\n",
      "                <transactionPricePerShare>\n",
      "                    <value>149.87</value>\n",
      "                </transactionPricePerShare>\n",
      "                <transactionAcquiredDisposedCode>\n",
      "                    <value>A</value>\n",
      "                </transactionAcquiredDisposedCode>\n",
      "            </transactionAmounts>\n",
      "            <postTransactionAmounts>\n",
      "                <sharesOwnedFollowingTransaction>\n",
      "                    <value>10609.2707</value>\n",
      "                </sharesOwnedFollowingTransaction>\n",
      "            </postTransactionAmounts>\n",
      "            <ownershipNature>\n",
      "                <directOrIndirectOwnership>\n",
      "                    <value>D</value>\n",
      "                </directOrIndirectOwnership>\n",
      "            </ownershipNature>\n",
      "        </nonDerivativeTransaction>\n",
      "        <nonDerivativeTransaction>\n",
      "            <securityTitle>\n",
      "                <value>Common Stock</value>\n",
      "            </securityTitle>\n",
      "            <transactionDate>\n",
      "                <value>2025-02-07</value>\n",
      "            </transactionDate>\n",
      "            <transactionCoding>\n",
      "                <transactionFormType>4</transactionFormType>\n",
      "                <transactionCode>F</transactionCode>\n",
      "                <equitySwapInvolved>0</equitySwapInvolved>\n",
      "            </transactionCoding>\n",
      "            <transactionTimeliness></transactionTimeliness>\n",
      "            <transactionAmounts>\n",
      "                <transactionShares>\n",
      "                    <value>1108</value>\n",
      "                </transactionShares>\n",
      "                <transactionPricePerShare>\n",
      "                    <value>149.84</value>\n",
      "                </transactionPricePerShare>\n",
      "                <transactionAcquiredDisposedCode>\n",
      "                    <value>D</value>\n",
      "                </transactionAcquiredDisposedCode>\n",
      "            </transactionAmounts>\n",
      "            <postTransactionAmounts>\n",
      "                <sharesOwnedFollowingTransaction>\n",
      "                    <value>9501.2707</value>\n",
      "                    <footnoteId id=\"F1\"/>\n",
      "                </sharesOwnedFollowingTransaction>\n",
      "            </postTransactionAmounts>\n",
      "            <ownershipNature>\n",
      "                <directOrIndirectOwnership>\n",
      "                    <value>D</value>\n",
      "                </directOrIndirectOwnership>\n",
      "            </ownershipNature>\n",
      "        </nonDerivativeTransaction>\n",
      "        <nonDerivativeHolding>\n",
      "            <securityTitle>\n",
      "                <value>Common Stock</value>\n",
      "            </securityTitle>\n",
      "            <postTransactionAmounts>\n",
      "                <sharesOwnedFollowingTransaction>\n",
      "                    <value>2209.385</value>\n",
      "                </sharesOwnedFollowingTransaction>\n",
      "            </postTransactionAmounts>\n",
      "            <ownershipNature>\n",
      "                <directOrIndirectOwnership>\n",
      "                    <value>D</value>\n",
      "                </directOrIndirectOwnership>\n",
      "            </ownershipNature>\n",
      "        </nonDerivativeHolding>\n",
      "    </nonDerivativeTable>\n",
      "\n",
      "    <derivativeTable>\n",
      "        <derivativeTransaction>\n",
      "            <securityTitle>\n",
      "                <value>Restricted Stock Units</value>\n",
      "            </securityTitle>\n",
      "            <conversionOrExercisePrice>\n",
      "                <footnoteId id=\"F2\"/>\n",
      "            </conversionOrExercisePrice>\n",
      "            <transactionDate>\n",
      "                <value>2025-02-07</value>\n",
      "            </transactionDate>\n",
      "            <transactionCoding>\n",
      "                <transactionFormType>4</transactionFormType>\n",
      "                <transactionCode>M</transactionCode>\n",
      "                <equitySwapInvolved>0</equitySwapInvolved>\n",
      "            </transactionCoding>\n",
      "            <transactionTimeliness></transactionTimeliness>\n",
      "            <transactionAmounts>\n",
      "                <transactionShares>\n",
      "                    <value>3569</value>\n",
      "                </transactionShares>\n",
      "                <transactionPricePerShare>\n",
      "                    <footnoteId id=\"F2\"/>\n",
      "                </transactionPricePerShare>\n",
      "                <transactionAcquiredDisposedCode>\n",
      "                    <value>D</value>\n",
      "                </transactionAcquiredDisposedCode>\n",
      "            </transactionAmounts>\n",
      "            <exerciseDate>\n",
      "                <footnoteId id=\"F3\"/>\n",
      "            </exerciseDate>\n",
      "            <expirationDate>\n",
      "                <footnoteId id=\"F3\"/>\n",
      "            </expirationDate>\n",
      "            <underlyingSecurity>\n",
      "                <underlyingSecurityTitle>\n",
      "                    <value>Common Stock</value>\n",
      "                </underlyingSecurityTitle>\n",
      "                <underlyingSecurityShares>\n",
      "                    <value>3569</value>\n",
      "                </underlyingSecurityShares>\n",
      "            </underlyingSecurity>\n",
      "            <postTransactionAmounts>\n",
      "                <sharesOwnedFollowingTransaction>\n",
      "                    <value>0</value>\n",
      "                </sharesOwnedFollowingTransaction>\n",
      "            </postTransactionAmounts>\n",
      "            <ownershipNature>\n",
      "                <directOrIndirectOwnership>\n",
      "                    <value>D</value>\n",
      "                </directOrIndirectOwnership>\n",
      "            </ownershipNature>\n",
      "        </derivativeTransaction>\n",
      "    </derivativeTable>\n",
      "\n",
      "    <footnotes>\n",
      "        <footnote id=\"F1\">Includes dividend share equivalents, accrued quarterly, pursuant to 3M's Deferred Compensation Plan.</footnote>\n",
      "        <footnote id=\"F2\">Each restricted stock unit represents a contingent right to receive one share of 3M common stock.</footnote>\n",
      "        <footnote id=\"F3\">The restricted stock units vest 100% three years from the grant date of February 8, 2022</footnote>\n",
      "    </footnotes>\n",
      "\n",
      "    <ownerSignature>\n",
      "        <signatureName>Patricia L. Meagher, attorney-in-fact for Beatriz Karina Chavez Rodriguez</signatureName>\n",
      "        <signatureDate>2025-02-18</signatureDate>\n",
      "    </ownerSignature>\n",
      "</ownershipDocument>\n",
      "</XML>\n",
      "</TEXT>\n",
      "</DOCUMENT>\n",
      "</SEC-DOCUMENT>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import httpx\n",
    "import hishel\n",
    "import os\n",
    "\n",
    "\n",
    "# Change this to the relevant output directory\n",
    "DATA_DIR = \"./data/insider_transactions\"\n",
    "if not os.path.exists(DATA_DIR):\n",
    "    os.makedirs(DATA_DIR)\n",
    "CACHE_DIR = os.path.join(DATA_DIR, \".httpx_cache\")\n",
    "FORM_4_DUMP_PATH = os.path.join(DATA_DIR, \"sec_form_4_dump.json\")\n",
    "\n",
    "\n",
    "transport = hishel.AsyncCacheTransport(\n",
    "    transport=httpx.AsyncHTTPTransport(),\n",
    "    storage=hishel.AsyncFileStorage(base_path=CACHE_DIR),\n",
    "    controller=hishel.Controller(force_cache=True),\n",
    ")\n",
    "\n",
    "# Use your own email below\n",
    "HEADERS = {\"User-Agent\": \"Your Name your_email@domain.com\"}\n",
    "http_client = httpx.AsyncClient(headers=HEADERS, timeout=60, transport=transport)\n",
    "example_url = \"https://www.sec.gov/Archives/edgar/data/66740/000112760225005147/0001127602-25-005147.txt\"\n",
    "r = await http_client.get(example_url)\n",
    "print(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Schema for Form 4 Extraction\n",
    "\n",
    "Let's define the data schema for the Form 4 extraction. We will use Pydantic to define our schema, and use LlamaExtract to extract the data from the Form 4 filings. Note that since we are using an LLM to extract the data, we don't need to worry about the exact layout/schema for the XML files. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pydantic import BaseModel, Field\n",
    "from typing import Dict, List, Optional\n",
    "\n",
    "\n",
    "class Address(BaseModel):\n",
    "    street1: str = Field(description=\"Business or owner street address line 1\")\n",
    "    street2: Optional[str] = Field(None, description=\"Street address line 2\")\n",
    "    city: str\n",
    "    state: str\n",
    "    zipCode: str\n",
    "\n",
    "\n",
    "class Issuer(BaseModel):\n",
    "    ticker: str = Field(description=\"Ticker of the issuer\")\n",
    "    name: str = Field(description=\"Name of the issuer\")\n",
    "    cik: str = Field(description=\"10-digit SEC CIK of the issuer\")\n",
    "    sic: Optional[str] = Field(\n",
    "        None, description=\"Standard Industrial Classification code\"\n",
    "    )\n",
    "    stateOfIncorporation: Optional[str]\n",
    "\n",
    "\n",
    "class ReportingOwner(BaseModel):\n",
    "    name: str = Field(description=\"Name of the reporting owner\")\n",
    "    cik: Optional[str] = Field(\n",
    "        None, description=\"CIK of the reporting owner (if available)\"\n",
    "    )\n",
    "    address: Address\n",
    "    relationship: Optional[str] = Field(\n",
    "        None, description=\"One of: Officer, Director, 10% owner, Affiliate, Other\"\n",
    "    )\n",
    "\n",
    "\n",
    "class UnderlyingSecurity(BaseModel):\n",
    "    title: str = Field(description=\"Title of the underlying security\")\n",
    "    conversionOrExercisePrice: Optional[float] = Field(None)\n",
    "    conversionOrExerciseDate: Optional[str] = Field(\n",
    "        None, description=\"Exercise date in ISO 8601 format (YYYY-MM-DD)\"\n",
    "    )\n",
    "    underlyingSecurityShares: Optional[int] = Field(\n",
    "        None, description=\"Number of shares underlying each derivative\"\n",
    "    )\n",
    "\n",
    "\n",
    "class NonDerivativeTransaction(BaseModel):\n",
    "    securityTitle: str = Field(description=\"Title of the security\")\n",
    "    transactionDate: str = Field(\n",
    "        description=\"Transaction date in ISO 8601 format (YYYY-MM-DD)\"\n",
    "    )\n",
    "    transactionCode: str = Field(\n",
    "        description=\"Code indicating transaction type (A, D, P, S, etc.)\"\n",
    "    )\n",
    "    transactionShares: float\n",
    "    transactionPricePerShare: float\n",
    "    totalTransactionValue: Optional[float] = Field(\n",
    "        None, description=\"transactionShares × transactionPricePerShare\"\n",
    "    )\n",
    "    ownershipNature: str = Field(description=\"e.g. Direct, Indirect\")\n",
    "    footnoteIds: Optional[List[str]] = Field(\n",
    "        None, description=\"References into the footnotes block\"\n",
    "    )\n",
    "\n",
    "\n",
    "class DerivativeTransaction(BaseModel):\n",
    "    securityTitle: str\n",
    "    transactionDate: str = Field(\n",
    "        description=\"Transaction date in ISO 8601 format (YYYY-MM-DD)\"\n",
    "    )\n",
    "    transactionCode: str\n",
    "    numberOfDerivativeSecurities: int\n",
    "    exercisePrice: Optional[float]\n",
    "    expirationDate: Optional[str] = Field(\n",
    "        description=\"Expiration date in ISO 8601 format (YYYY-MM-DD)\"\n",
    "    )\n",
    "    underlyingSecurity: UnderlyingSecurity\n",
    "    directOrIndirect: str = Field(description=\"Direct or Indirect\")\n",
    "    footnoteIds: Optional[List[str]]\n",
    "\n",
    "\n",
    "class Footnote(BaseModel):\n",
    "    id: str = Field(description=\"Footnote identifier\")\n",
    "    text: str = Field(description=\"Full footnote text\")\n",
    "\n",
    "\n",
    "class Form4Metadata(BaseModel):\n",
    "    cik: str = Field(description=\"CIK of the filer\")\n",
    "    accessionNumber: str\n",
    "    primaryDocument: str = Field(description=\"Filename of the primary XML/TXT\")\n",
    "    filingDate: str = Field(description=\"Filing date in ISO 8601 format (YYYY-MM-DD)\")\n",
    "    reportDate: str = Field(\n",
    "        description=\"Period of report date in ISO 8601 format (YYYY-MM-DD)\"\n",
    "    )\n",
    "\n",
    "\n",
    "class Form4(BaseModel):\n",
    "    metadata: Form4Metadata\n",
    "    issuer: Issuer\n",
    "    reportingOwners: List[ReportingOwner]\n",
    "    nonDerivativeTransactions: List[NonDerivativeTransaction]\n",
    "    derivativeTransactions: List[DerivativeTransaction]\n",
    "    footnotes: Optional[List[Footnote]] = Field(default=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the LlamaExtract Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dotenv import load_dotenv\n",
    "from llama_cloud_services import LlamaExtract\n",
    "\n",
    "\n",
    "# Load environment variables (put LLAMA_CLOUD_API_KEY in your .env file)\n",
    "load_dotenv(override=True)\n",
    "\n",
    "# Optionally, add your project id/organization id\n",
    "extract = LlamaExtract(show_progress=False, check_interval=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_cloud.core.api_error import ApiError\n",
    "\n",
    "try:\n",
    "    existing_agent = extract.get_agent(name=\"Form4Extractor\")\n",
    "    if existing_agent:\n",
    "        # Deletion can take some time since all underlying files will be purged\n",
    "        extract.delete_agent(existing_agent.id)\n",
    "except ApiError as e:\n",
    "    if e.status_code == 404:\n",
    "        pass\n",
    "    else:\n",
    "        raise\n",
    "\n",
    "agent = extract.create_agent(\"Form4Extractor\", data_schema=Form4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example Extraction\n",
    "\n",
    "We will write some utility functions to get the CIK mapping and the SEC Form 4 URLs for all the companies in the Dow Jones Industrial Average. \n",
    "\n",
    "⚠ Note that we need to limit ourselves to within 10reqs/second when hitting the SEC data so as not to get IP banned. We will use AsyncLimited for aiolimiter in our workflow for this. \n",
    "\n",
    "To do this, we will \n",
    " - use `hishel` to cache the responses from the SEC website to avoid being rate limited or blocked, \n",
    " - use `aiolimiter` to limit the rate of requests to the SEC website, and \n",
    " - pre-populate a cache of the SEC Form 4 filings into a data dump that can be shared with a wider group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: aiolimiter in /Users/neerajprad/workspace/llama_cloud_services/.venv/lib/python3.11/site-packages (1.2.1)\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.1.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install aiolimiter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "from aiolimiter import AsyncLimiter\n",
    "from collections import defaultdict\n",
    "\n",
    "\n",
    "rate_limiter = AsyncLimiter(max_rate=5, time_period=1)\n",
    "\n",
    "\n",
    "async def get_cik_mapping(tickers: List[str]) -> Dict[str, str]:\n",
    "    if os.path.exists(\"./data/insider_transactions/cik_mapping.json\"):\n",
    "        with open(\"./data/insider_transactions/cik_mapping.json\", \"r\") as f:\n",
    "            return json.load(f)\n",
    "    mapping_resp = await http_client.get(\n",
    "        \"https://www.sec.gov/files/company_tickers.json\",\n",
    "        headers=HEADERS,\n",
    "    )\n",
    "    mapping_resp.raise_for_status()\n",
    "    mapping = mapping_resp.json()\n",
    "    ticker_to_cik = {v[\"ticker\"]: str(v[\"cik_str\"]).zfill(10) for v in mapping.values()}\n",
    "    with open(\"./data/insider_transactions/cik_mapping.json\", \"w\") as f:\n",
    "        json.dump(ticker_to_cik, f, indent=2)\n",
    "    return {k: v for k, v in ticker_to_cik.items() if k in tickers}\n",
    "\n",
    "\n",
    "async def get_sec_form4_urls(\n",
    "    tickers: List[str], start_date: str, end_date: str\n",
    ") -> List[str]:\n",
    "    ticker_to_cik = await get_cik_mapping(tickers)\n",
    "    urls = defaultdict(list)\n",
    "    for ticker in tickers:\n",
    "        cik = ticker_to_cik[ticker]\n",
    "        url = f\"https://data.sec.gov/submissions/CIK{cik}.json\"\n",
    "        async with rate_limiter:\n",
    "            resp = await http_client.get(url)\n",
    "            resp.raise_for_status()\n",
    "\n",
    "        recent = resp.json().get(\"filings\", {}).get(\"recent\", {})\n",
    "        forms = recent.get(\"form\", [])\n",
    "        dates = recent.get(\"filingDate\", [])\n",
    "        accession = recent.get(\"accessionNumber\", [])\n",
    "        for f, d, acc in zip(forms, dates, accession):\n",
    "            if f == \"4\" and start_date <= d <= end_date:\n",
    "                idx = acc.replace(\"-\", \"\")\n",
    "                urls[ticker].append(\n",
    "                    f\"https://www.sec.gov/Archives/edgar/data/{int(cik)}/{idx}/{acc}.txt\",\n",
    "                )\n",
    "    return urls\n",
    "\n",
    "\n",
    "async def get_sec_form4(url):\n",
    "    form_4_cached = None\n",
    "    if os.path.exists(FORM_4_DUMP_PATH):\n",
    "        with open(FORM_4_DUMP_PATH, \"r\") as f:\n",
    "            form_4_cached = json.load(f)\n",
    "    accession_no = url.split(\"/\")[-1].split(\".\")[0]\n",
    "    if form_4_cached and accession_no in form_4_cached:\n",
    "        return form_4_cached[accession_no]\n",
    "    async with rate_limiter:\n",
    "        resp = await http_client.get(url)\n",
    "        resp.raise_for_status()\n",
    "    return resp.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(list,\n",
       "            {'AAPL': ['https://www.sec.gov/Archives/edgar/data/320193/000032019325000036/0000320193-25-000036.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019325000035/0000320193-25-000035.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019325000034/0000320193-25-000034.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019325000033/0000320193-25-000033.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019325000032/0000320193-25-000032.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019325000031/0000320193-25-000031.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019325000030/0000320193-25-000030.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019325000022/0000320193-25-000022.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019325000021/0000320193-25-000021.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019325000020/0000320193-25-000020.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019325000019/0000320193-25-000019.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019325000018/0000320193-25-000018.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019325000017/0000320193-25-000017.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019325000016/0000320193-25-000016.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000132/0000320193-24-000132.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000130/0000320193-24-000130.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000129/0000320193-24-000129.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000126/0000320193-24-000126.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000116/0000320193-24-000116.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000114/0000320193-24-000114.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000112/0000320193-24-000112.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000111/0000320193-24-000111.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000110/0000320193-24-000110.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000109/0000320193-24-000109.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000108/0000320193-24-000108.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000101/0000320193-24-000101.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000100/0000320193-24-000100.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000099/0000320193-24-000099.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000098/0000320193-24-000098.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000097/0000320193-24-000097.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000096/0000320193-24-000096.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000088/0000320193-24-000088.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000086/0000320193-24-000086.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000084/0000320193-24-000084.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000075/0000320193-24-000075.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000073/0000320193-24-000073.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000071/0000320193-24-000071.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000058/0000320193-24-000058.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000056/0000320193-24-000056.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000055/0000320193-24-000055.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000052/0000320193-24-000052.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000051/0000320193-24-000051.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000050/0000320193-24-000050.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000049/0000320193-24-000049.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000048/0000320193-24-000048.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000039/0000320193-24-000039.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000038/0000320193-24-000038.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000037/0000320193-24-000037.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000036/0000320193-24-000036.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000035/0000320193-24-000035.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000034/0000320193-24-000034.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000033/0000320193-24-000033.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000022/0000320193-24-000022.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000021/0000320193-24-000021.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000020/0000320193-24-000020.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000019/0000320193-24-000019.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000018/0000320193-24-000018.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000017/0000320193-24-000017.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000016/0000320193-24-000016.txt',\n",
       "              'https://www.sec.gov/Archives/edgar/data/320193/000032019324000015/0000320193-24-000015.txt']})"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "form_4_urls = await get_sec_form4_urls([\"AAPL\"], \"2024-01-01\", \"2025-03-31\")\n",
    "form_4_urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "async def prepopulate_form_4_cache(tickers, start_date, end_date):\n",
    "    form_4_dump = await http_client.get(\n",
    "        \"http://llamaindex-public-demo.s3.amazonaws.com/sec_form_4_dump.json\"\n",
    "    )\n",
    "    with open(FORM_4_DUMP_PATH, \"w\") as f:\n",
    "        f.write(form_4_dump.text)\n",
    "\n",
    "    # Seeding for the first time:\n",
    "    #\n",
    "    # cache = {}\n",
    "    # for ticker in tickers:\n",
    "    #     form_4_urls = await get_sec_form4_urls(tickers, start_date, end_date)\n",
    "    #     print(ticker)\n",
    "    #     for url in form_4_urls[ticker]:\n",
    "    #         form_4 = await get_sec_form4(url)\n",
    "    #         cache[url.split(\"/\")[-1].split(\".\")[0]] = form_4\n",
    "    # with open(FORM_4_DUMP_PATH, \"w\") as f:\n",
    "    #     json.dump(cache, f, indent=2)\n",
    "\n",
    "\n",
    "START_DATE = \"2024-01-01\"\n",
    "END_DATE = \"2025-03-31\"\n",
    "\n",
    "await prepopulate_form_4_cache(DJIA_TICKERS, START_DATE, END_DATE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_form_4 = await get_sec_form4(\n",
    "    \"https://www.sec.gov/Archives/edgar/data/320193/000032019325000036/0000320193-25-000036.txt\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'metadata': {'cik': '0001059235',\n",
       "  'accessionNumber': '0000320193-25-000036',\n",
       "  'primaryDocument': 'wk-form4_1740699336.xml',\n",
       "  'filingDate': '2025-02-27',\n",
       "  'reportDate': '2025-02-25'},\n",
       " 'issuer': {'ticker': 'AAPL',\n",
       "  'name': 'Apple Inc.',\n",
       "  'cik': '0000320193',\n",
       "  'sic': '3571',\n",
       "  'stateOfIncorporation': 'CA'},\n",
       " 'reportingOwners': [{'name': 'WAGNER SUSAN',\n",
       "   'cik': '0001059235',\n",
       "   'address': {'street1': 'ONE APPLE PARK WAY',\n",
       "    'street2': None,\n",
       "    'city': 'CUPERTINO',\n",
       "    'state': 'CA',\n",
       "    'zipCode': '95014'},\n",
       "   'relationship': 'Director'}],\n",
       " 'nonDerivativeTransactions': [],\n",
       " 'derivativeTransactions': [{'securityTitle': 'Restricted Stock Unit',\n",
       "   'transactionDate': '2025-02-25',\n",
       "   'transactionCode': 'A',\n",
       "   'numberOfDerivativeSecurities': 1255,\n",
       "   'exercisePrice': None,\n",
       "   'expirationDate': '2026-02-01',\n",
       "   'underlyingSecurity': {'title': 'Common Stock',\n",
       "    'conversionOrExercisePrice': None,\n",
       "    'conversionOrExerciseDate': '2026-02-01',\n",
       "    'underlyingSecurityShares': 1255},\n",
       "   'directOrIndirect': 'D',\n",
       "   'footnoteIds': ['F1', 'F2', 'F3']}],\n",
       " 'footnotes': [{'id': 'F1',\n",
       "   'text': 'Automatic grant pursuant to the Apple Inc. Non-Employee Director Stock Plan.'},\n",
       "  {'id': 'F2',\n",
       "   'text': 'Each restricted stock unit represents the right to receive, at settlement, one share of common stock.'},\n",
       "  {'id': 'F3',\n",
       "   'text': '100% of these restricted stock units are scheduled to vest on February 1, 2026, assuming continued service through the vesting date.'}]}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from llama_cloud_services.extract import SourceText\n",
    "\n",
    "agent.extract(SourceText(text_content=example_form_4)).data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Workflow for Extracting Insider Transactions from SEC Form 4\n",
    "\n",
    "Now that we have defined the LlamaExtract agent, we can create a workflow to scalably extract the insider transactions from SEC Form 4 filings for all the companies in the Dow Jones Industrial Average. We will use all the filings during 2024 and the first quarter of 2025.\n",
    "\n",
    "\n",
    "### Exporting to Pandas DataFame for Analysis\n",
    "\n",
    "Once we have the data extracted, we can create a Pandas DataFrame with the relevant subset of fields that we can use for analysis. In `forms_to_dataframe`, we will need to map the nested JSON data extracted using our schema above to a flat Pandas DataFrame for analysis.\n",
    "\n",
    "**Workflow diagram:**\n",
    "\n",
    "![Workflow Diagram](./data/insider_transactions/workflow-diag.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.workflow import (\n",
    "    Context,\n",
    "    Workflow,\n",
    "    Event,\n",
    "    StartEvent,\n",
    "    StopEvent,\n",
    "    step,\n",
    ")\n",
    "from llama_index.core.workflow.retry_policy import ConstantDelayRetryPolicy\n",
    "\n",
    "\n",
    "class SECForm4URL(Event):\n",
    "    \"Form 4 URL to extract from\"\n",
    "    ticker: str\n",
    "    url: str\n",
    "\n",
    "\n",
    "class ExtractedData(Event):\n",
    "    \"Data extracted from SEC Form 4\"\n",
    "    extracted: dict\n",
    "\n",
    "\n",
    "class ExtractedDataCollection(Event):\n",
    "    \"Collection of extracted data for all tickers\"\n",
    "    collection: List[dict]\n",
    "\n",
    "\n",
    "class Form4Extraction(Workflow):\n",
    "    def __init__(self, start_date: str, end_date: str, verbose=False, timeout=1800):\n",
    "        super().__init__(verbose=verbose, timeout=timeout)\n",
    "        self.start_date = start_date\n",
    "        self.end_date = end_date\n",
    "        self.limiter = AsyncLimiter(max_rate=9, time_period=1)\n",
    "        self.result = None\n",
    "\n",
    "    @step\n",
    "    async def get_djia_companies(self, ctx: Context, ev: StartEvent) -> SECForm4URL:\n",
    "        \"\"\"\n",
    "        Fetch all Form 4 URLs for companies in the DJIA index\n",
    "        between (start_date, end_date).\n",
    "        \"\"\"\n",
    "        url = \"https://en.wikipedia.org/wiki/Dow_Jones_Industrial_Average\"\n",
    "        parsed = pd.read_html(url, header=0)\n",
    "        dow = next(t for t in parsed if \"Symbol\" in t.columns)\n",
    "        tickers = dow[\"Symbol\"].tolist()\n",
    "        if self._verbose:\n",
    "            print(f\"Number of companies to extract: {len(tickers)}\")\n",
    "        result = []\n",
    "        ticker_urls = await get_sec_form4_urls(tickers, self.start_date, self.end_date)\n",
    "        for ticker, urls in ticker_urls.items():\n",
    "            for url in urls:\n",
    "                result.append(SECForm4URL(ticker=ticker, url=url))\n",
    "        if self._verbose:\n",
    "            print(f\"Number of forms to extract: {len(result)}\")\n",
    "        await ctx.set(\"num_forms\", len(result))\n",
    "        for url in result:\n",
    "            ctx.send_event(url)\n",
    "\n",
    "    @step(\n",
    "        num_workers=10,\n",
    "        retry_policy=ConstantDelayRetryPolicy(maximum_attempts=3, delay=10),\n",
    "    )\n",
    "    async def fetch_sec_form4_filings(self, ev: SECForm4URL) -> ExtractedData:\n",
    "        \"\"\"\n",
    "        Download form 4 from the SEC website, and use LlamaExtract agent defined earlier\n",
    "        to extract the data from the form.\n",
    "        \"\"\"\n",
    "        async with self.limiter:\n",
    "            form_4 = await get_sec_form4(ev.url)\n",
    "        extracted = await agent.aextract(\n",
    "            SourceText(text_content=f\"ticker: {ev.ticker}\\n\\nform 4: {form_4}\")\n",
    "        )\n",
    "        return ExtractedData(extracted=extracted.data)\n",
    "\n",
    "    @step\n",
    "    async def collect_results(\n",
    "        self, ctx: Context, ev: ExtractedData\n",
    "    ) -> ExtractedDataCollection | None:\n",
    "        \"\"\"\n",
    "        Collect all the data from the SEC forms into results.\n",
    "        \"\"\"\n",
    "        num_forms = await ctx.get(\"num_forms\")\n",
    "        results = ctx.collect_events(ev, [ExtractedData] * num_forms)\n",
    "        if results is None:\n",
    "            return\n",
    "        self.result = [r.extracted for r in results]\n",
    "        if self._verbose:\n",
    "            print(f\"{len(self.result)} results extracted.\")\n",
    "        return ExtractedDataCollection(collection=self.result)\n",
    "\n",
    "    @step\n",
    "    async def forms_to_dataframe(\n",
    "        self, ctx: Context, ev: ExtractedDataCollection\n",
    "    ) -> StopEvent:\n",
    "        \"\"\"\n",
    "        Convert the extracted data into a Pandas DataFrame.\n",
    "        \"\"\"\n",
    "        # 1) map SEC transaction codes to human labels\n",
    "        # Add any other codes you need to consider in your analysis...\n",
    "        code_map = {\n",
    "            \"S\": \"Sale\",\n",
    "            \"P\": \"Proposed Sale\",\n",
    "            \"A\": \"Acquisition\",\n",
    "            \"D\": \"Disposition\",\n",
    "        }\n",
    "        records = []\n",
    "        for form in ev.collection:\n",
    "            filing_date = form[\"metadata\"][\"filingDate\"]\n",
    "            # loop each reporting owner (usually one)\n",
    "            for owner in form.get(\"reportingOwners\", []):\n",
    "                # non-derivative transactions\n",
    "                for tx in form.get(\"nonDerivativeTransactions\", []) or []:\n",
    "                    records.append(\n",
    "                        {\n",
    "                            \"Ticker\": form[\"issuer\"][\"ticker\"],\n",
    "                            \"Company Name\": form[\"issuer\"][\"name\"],\n",
    "                            \"Insider Name\": owner[\"name\"],\n",
    "                            \"Relationship\": owner.get(\"relationship\"),\n",
    "                            \"Transaction Date\": pd.to_datetime(tx[\"transactionDate\"]),\n",
    "                            \"Transaction\": code_map.get(\n",
    "                                tx[\"transactionCode\"], tx[\"transactionCode\"]\n",
    "                            ),\n",
    "                            \"Cost\": tx[\"transactionPricePerShare\"],\n",
    "                            \"#Shares\": tx[\"transactionShares\"],\n",
    "                            \"Value ($)\": tx.get(\n",
    "                                \"totalTransactionValue\",\n",
    "                                tx[\"transactionShares\"]\n",
    "                                * tx[\"transactionPricePerShare\"],\n",
    "                            ),\n",
    "                            \"#Shares Total\": tx.get(\"postTransactionShares\"),\n",
    "                            \"Filing Date\": pd.to_datetime(filing_date),\n",
    "                        }\n",
    "                    )\n",
    "\n",
    "                # derivative transactions\n",
    "                for tx in form.get(\"derivativeTransactions\", []) or []:\n",
    "                    records.append(\n",
    "                        {\n",
    "                            \"Ticker\": form[\"issuer\"][\"ticker\"],\n",
    "                            \"Company Name\": form[\"issuer\"][\"name\"],\n",
    "                            \"Insider Name\": owner[\"name\"],\n",
    "                            \"Relationship\": owner.get(\"relationship\"),\n",
    "                            \"Transaction Date\": pd.to_datetime(tx[\"transactionDate\"]),\n",
    "                            \"Transaction\": code_map.get(\n",
    "                                tx[\"transactionCode\"], tx[\"transactionCode\"]\n",
    "                            ),\n",
    "                            \"Cost\": tx.get(\n",
    "                                \"transactionPricePerShare\", tx.get(\"exercisePrice\")\n",
    "                            ),\n",
    "                            \"#Shares\": tx.get(\"numberOfDerivativeSecurities\"),\n",
    "                            \"Value ($)\": None,\n",
    "                            \"#Shares Total\": None,\n",
    "                            \"Filing Date\": pd.to_datetime(filing_date),\n",
    "                        }\n",
    "                    )\n",
    "\n",
    "        self.df = pd.DataFrame(records)\n",
    "        self.df = self.df.sort_values(by=[\"Ticker\", \"Transaction Date\"])\n",
    "        return StopEvent()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running step get_djia_companies\n",
      "Number of companies to extract: 30\n",
      "Number of forms to extract: 259\n",
      "Step get_djia_companies produced no event\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step fetch_sec_form4_filings\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step fetch_sec_form4_filings\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step collect_results\n",
      "Step collect_results produced no event\n",
      "Step fetch_sec_form4_filings produced event ExtractedData\n",
      "Running step collect_results\n",
      "259 results extracted.\n",
      "Step collect_results produced event ExtractedDataCollection\n",
      "Running step forms_to_dataframe\n",
      "Step forms_to_dataframe produced event StopEvent\n"
     ]
    }
   ],
   "source": [
    "extraction = Form4Extraction(\n",
    "    start_date=\"2025-03-01\", end_date=\"2025-03-31\", verbose=True\n",
    ")\n",
    "await extraction.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Ticker</th>\n",
       "      <th>Company Name</th>\n",
       "      <th>Insider Name</th>\n",
       "      <th>Relationship</th>\n",
       "      <th>Transaction Date</th>\n",
       "      <th>Transaction</th>\n",
       "      <th>Cost</th>\n",
       "      <th>#Shares</th>\n",
       "      <th>Value ($)</th>\n",
       "      <th>#Shares Total</th>\n",
       "      <th>Filing Date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>AMGN</td>\n",
       "      <td>AMGEN INC</td>\n",
       "      <td>Santos Esteban</td>\n",
       "      <td>Officer</td>\n",
       "      <td>2025-03-03</td>\n",
       "      <td>Acquisition</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>6792.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>None</td>\n",
       "      <td>2025-03-05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>AMGN</td>\n",
       "      <td>AMGEN INC</td>\n",
       "      <td>Miller Derek</td>\n",
       "      <td>Officer</td>\n",
       "      <td>2025-03-03</td>\n",
       "      <td>Acquisition</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>2005.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>None</td>\n",
       "      <td>2025-03-05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>AMGN</td>\n",
       "      <td>AMGEN INC</td>\n",
       "      <td>REESE DAVID M</td>\n",
       "      <td>Officer</td>\n",
       "      <td>2025-03-03</td>\n",
       "      <td>Acquisition</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>7277.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>None</td>\n",
       "      <td>2025-03-05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>AMGN</td>\n",
       "      <td>AMGEN INC</td>\n",
       "      <td>Griffith Peter H.</td>\n",
       "      <td>Officer</td>\n",
       "      <td>2025-03-03</td>\n",
       "      <td>Acquisition</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>7277.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>None</td>\n",
       "      <td>2025-03-05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>AMGN</td>\n",
       "      <td>AMGEN INC</td>\n",
       "      <td>Grygiel Nancy A.</td>\n",
       "      <td>Officer</td>\n",
       "      <td>2025-03-03</td>\n",
       "      <td>Acquisition</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>1503.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>None</td>\n",
       "      <td>2025-03-05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>446</th>\n",
       "      <td>WMT</td>\n",
       "      <td>Walmart Inc.</td>\n",
       "      <td>Penner Gregory Boyd</td>\n",
       "      <td>Director</td>\n",
       "      <td>2025-03-26</td>\n",
       "      <td>J</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>27000.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>None</td>\n",
       "      <td>2025-03-28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>457</th>\n",
       "      <td>WMT</td>\n",
       "      <td>Walmart Inc.</td>\n",
       "      <td>Walton Family Holdings Trust</td>\n",
       "      <td>10% owner</td>\n",
       "      <td>2025-03-26</td>\n",
       "      <td>J</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>297000.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>None</td>\n",
       "      <td>2025-03-28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>440</th>\n",
       "      <td>WMT</td>\n",
       "      <td>Walmart Inc.</td>\n",
       "      <td>McMillon C Douglas</td>\n",
       "      <td>Officer, Director</td>\n",
       "      <td>2025-03-27</td>\n",
       "      <td>Sale</td>\n",
       "      <td>85.6333</td>\n",
       "      <td>29124.0</td>\n",
       "      <td>2.495473e+06</td>\n",
       "      <td>None</td>\n",
       "      <td>2025-03-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>445</th>\n",
       "      <td>WMT</td>\n",
       "      <td>Walmart Inc.</td>\n",
       "      <td>Walton Steuart L</td>\n",
       "      <td>Director</td>\n",
       "      <td>2025-03-28</td>\n",
       "      <td>G</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>27000.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>None</td>\n",
       "      <td>2025-03-28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>447</th>\n",
       "      <td>WMT</td>\n",
       "      <td>Walmart Inc.</td>\n",
       "      <td>Penner Gregory Boyd</td>\n",
       "      <td>Director</td>\n",
       "      <td>2025-03-28</td>\n",
       "      <td>G</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>27000.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>None</td>\n",
       "      <td>2025-03-28</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>506 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    Ticker  Company Name                  Insider Name       Relationship  \\\n",
       "8     AMGN     AMGEN INC                Santos Esteban            Officer   \n",
       "9     AMGN     AMGEN INC                  Miller Derek            Officer   \n",
       "10    AMGN     AMGEN INC                 REESE DAVID M            Officer   \n",
       "13    AMGN     AMGEN INC             Griffith Peter H.            Officer   \n",
       "14    AMGN     AMGEN INC              Grygiel Nancy A.            Officer   \n",
       "..     ...           ...                           ...                ...   \n",
       "446    WMT  Walmart Inc.           Penner Gregory Boyd           Director   \n",
       "457    WMT  Walmart Inc.  Walton Family Holdings Trust          10% owner   \n",
       "440    WMT  Walmart Inc.            McMillon C Douglas  Officer, Director   \n",
       "445    WMT  Walmart Inc.              Walton Steuart L           Director   \n",
       "447    WMT  Walmart Inc.           Penner Gregory Boyd           Director   \n",
       "\n",
       "    Transaction Date  Transaction     Cost   #Shares     Value ($)  \\\n",
       "8         2025-03-03  Acquisition   0.0000    6792.0  0.000000e+00   \n",
       "9         2025-03-03  Acquisition   0.0000    2005.0  0.000000e+00   \n",
       "10        2025-03-03  Acquisition   0.0000    7277.0  0.000000e+00   \n",
       "13        2025-03-03  Acquisition   0.0000    7277.0  0.000000e+00   \n",
       "14        2025-03-03  Acquisition   0.0000    1503.0  0.000000e+00   \n",
       "..               ...          ...      ...       ...           ...   \n",
       "446       2025-03-26            J   0.0000   27000.0  0.000000e+00   \n",
       "457       2025-03-26            J   0.0000  297000.0  0.000000e+00   \n",
       "440       2025-03-27         Sale  85.6333   29124.0  2.495473e+06   \n",
       "445       2025-03-28            G   0.0000   27000.0  0.000000e+00   \n",
       "447       2025-03-28            G   0.0000   27000.0  0.000000e+00   \n",
       "\n",
       "    #Shares Total Filing Date  \n",
       "8            None  2025-03-05  \n",
       "9            None  2025-03-05  \n",
       "10           None  2025-03-05  \n",
       "13           None  2025-03-05  \n",
       "14           None  2025-03-05  \n",
       "..            ...         ...  \n",
       "446          None  2025-03-28  \n",
       "457          None  2025-03-28  \n",
       "440          None  2025-03-31  \n",
       "445          None  2025-03-28  \n",
       "447          None  2025-03-28  \n",
       "\n",
       "[506 rows x 11 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extraction.df"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "llama-cloud-services",
   "language": "python",
   "name": "llama-cloud-services"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
